The present disclosure relates to testing coverage in general, and to functional coverage, in particular.
Computerized devices control almost every aspect of our life—from writing documents to controlling traffic lights. However, computerized devices are bug-prone, and thus require a testing phase in which the bugs should be discovered. The testing phase is considered one of the most difficult tasks in designing a computerized device. The cost of not discovering a bug may be enormous, as the consequences of the bug may be disastrous. For example, a bug may cause the injury of a person relying on the designated behavior of the computerized device. Additionally, a bug in hardware or firmware may be expensive to fix, as patching it requires call-back of the computerized device. Hence, many developers of computerized devices invest a substantial portion of the development cycle to discover erroneous behaviors of the computerized device.
During the testing phase, a sample of all possible behaviors of the computerized device is inspected. Coverage tools for checking software provide a measure of how well the software being evaluated has been exercised during testing and thereby give a level of assurance that the software is of high quality. A coverage model defines elements of the target system, and enables determination of whether a coverage task, associated with an element of the target system, has been executed, operated or the like.
There are a number of types of coverage models known in the art, such as statement coverage, line coverage, condition coverage, path coverage, method coverage, and the like. One additional coverage model is functional coverage. Functional coverage defines functional elements of the target system. A functional element defines a functional state, behavior or the like of the target system. Functional coverage may be used to measure amount, portion or a similar metric of tests that examined predetermined functional behaviors. Once functional coverage is measured, quality assurance (QA) personnel may design additional tests to examine untested functional elements.
Traditionally, the functional coverage model is defined by a QA staff member, an engineer, a programmer or the like. The functional coverage model comprises a set of attributes, each having a corresponding set of possible values. The functional coverage may define a set of coverage tasks, each comprising a combination of values of the different attributes. In some cases, each pair, triplet, quadruplet and the like of values of two attributes defines a functional element, also referred to as a coverage task. As opposed to other coverage methods, the functional coverage model may be used for determining whether a predetermined behavior was covered. A coverage task of a functional coverage model may describe the expected behavior of the system. For example, a functional coverage model may list important recovery states that should be tested. A coverage task of a code-based coverage model, on the other hand, indicates whether the code element listed in the task (e.g., a line in a source file) has been visited.
Functional coverage checks coverage of coverage tasks in a trace. The trace comprises a plurality of entries, each entry indicative of a coverage task that was covered by a target device, system, program or the like (referred to hereinbelow as a target system). For example, each entry may describe a state of a transaction performed by the target system. As another example, an entry may describe a state of one or more concurrent entities of the target system (such as software services, software threads, hardware threads, cores of a CPU, processors, or the like). The trace may be generated during execution of the target system. The entry may comprise values of functional coverage attributes. In some cases, some entries may not indicate a value of each attribute, but rather a portion thereof.
The functional coverage model may further comprise a set of restrictions defining a series of values of different attributes that may not appear together. For example, consider a functional coverage defining two attributes: ACTION and USER. The ACTION attribute may be each of the following values: RETRIEVE, STORE, and MODIFY PERMISSION. The USER attribute may be each of the following values: ADMIN, USER, GUEST. In some cases, a guest user cannot modify permission. A restriction may be defined to indicate that the couple (GUEST, MODIFY PERMISSION) is not a valid couple. The fact that a trace does not comprise an entry covering a coverage task that includes the couple does not affect the functional coverage. In other words, all possible coverage tasks—which together form the maximal possible coverage in respect to a functional coverage—do not include any coverage task that comprises the restricted couple.
Functional coverage may be measured in respect to a “white box” target system or to a “black box” target system. For example, the implementation of the target system may not be available during execution, and thus be referred to as “black box”. Using the input and output of the target system, such as input signals and logged events, a functional state of the target system may be evaluated.